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ABSTRACT 

The three diagnostic dimensions of the Concerns Based 
Adoption Model (CBAM) are tools that can evaluate the degree of new 
program implementation at the classroom level. Implementation success 
is a function of r«se/nonuse, appropriate/inappropriate practice, and 
user concerns about the innovation. These dimensions: Levels of Use 
(LoU) of the Innovation, Innovation Configurations (IC) and Stages of 
Concern (SoC) about the Innovation can measure and rank progress for 
individuals or groups, be combined for a composite ranking, and be 
used for exploring the relationship between implementation success 
and change process variables. These procedures were applied to the 
Principal-Teacher Interaction Study to assess implementation success 
across nine elementary schools in Colorado, Florida, and California. 
Study sites varied by principal's approach to facilitation, year of 
implementation, and innovation being implemented, Throughout the 
1980-81 school year, data were collected from bi-weekly phone calls 
to principals and four on-site visits. Teacher data were collected 
using the CBAM diagnostic dimensions. Rankings were done during three 
3-day meetings by research staff and school district representatives. 
Results suggest close monitoring feed back and more interventions 
will improve implementation success. Principal change facilitator 
styl" significantly influenced teacher classroom practice. The 
appendix details the three diagnostic dimensions of the CBAM. (BS) 
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School improvement efforts, to a large degree, are based on 
Implementation of new programs, and change 1n curriculum and teaching 
practices (I.e. Innovations). The Issue of Innovation Implementation success 
therefore 1s a crucial one when considering school Improvement. School 
Improvement 1s not only related to the quality of the educational innovation, 
but also to how well and to what degree the Innovation 1s implemented. To 
state it simply, a very high quality program implemented poorly is not likely 
to produce the desired outcomes. 

In the past, there have been numerous ways to evaluate the effects of 
educational Innovations, but very few tools for systematically evaluating the 
degree of , implementation at the classroom level. The three diagnostic 
dimensions of the Concerns Based Adoption Model (Hall, Wallace & Doss*tt, 
1973) are such tools. These dimensions: stages of Concern, Levels of Use and 
Innovation Configurations have been used to assess Implementation success 1n a 
recent study of school Improvement efforts. 
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This paper will describe a process that was used to assess implementation 
success across nine schools 1n three widely separated school districts and 
will report selected findings about the relationship between change process 
variables and Implementation success. This procedure can be used to compare 
Implementation success at a given point in time across sites and across 
Innovations and to correlate implementation success with other change process 
variables. It 1s primarily useful for comparing the implementation success of 
two or more Innovations or of one or more Innovations being implemented 1n two 
or more settings. Therefore, the procedure described in this paper has 
application value to researchers, evaluators and others concerned with the 
study of school improvement. It is especially valuable when there are 
multiple sites and/or multiple implementations. For these reasons, the 
procedure appears to be a highly promising technique. • 

Defining Implementation Success 

Implementation success 1s an elusive concept. Often success seems to be 
based more on who 1s making the judgement than on any characteristics of the 
implementation effort, user practice or change process effects. One view of 
successful Implementation 1s based solely upon evidence that the box of 
Innovation resources 1s 1n the classroom. From another view, successful 
Implementation would not be declared until user behaviors with the innovation 
resources have been extensively documented. 

Most frequently Implementation success is addressed 1n terms of use of a 
single Innovation. Making comparative judgements across two or more 
Innovations has been problematic and has often been equated to comparing 
apples and oranges. However, a procedure has been developed and tested for 
interpreting implementation success that 1s sensitive to differences in 



Innovations and to the inherent problems related to comparisons across 
innovations. After all, apples and oranges are both fruit and can be compared 
on that dimension. 

Increasingly, factors related to Implementation are being considered in 
planning, facilitating and evaluating change efforts. Simply placing the 
innovation 1n the hands of prospective users and then corducting pre/post 
tests of differences 1n client outcomes no longer mak sense (Charters & 
Jones, 1973). Change 1s a process, not an event. This assertion is true of 
Institutions as well as the individual members who comprise them. It has been 
learned that use is not dlchotomous in a change process rather there are 
different Levels- of Use (Hall, Loucks, Rutherford & Newlove, 1976). Further 
innovations are not always used as the developer envisioned (Berman & 
McLaughlin, 1978) and each user may use a different operational form or 
"configuration" of the Innovation (Hall * Loucks, 1978). For these and other 
reasons implementation has to be considered as a distinct phase and phenomenon 
1n the life of a change/Improvement process. 

A given assumption 1n any discussion of implementation success 1s that 
some forms of use of an Innovation are more desirable than others. Who makes 
this decision is different for different projects and must be determined 
specifically for each new project. However, 1n order to judge implementation 
success someone or group has to specify which approaches are more and less 
desirable. Although this seems to be an obvious step, all too frequently 
these criteria are not Identified or are left 1n terms so vague that any type 
of use at the school or classroom level 1s assumed to be acceptable. 

Note that the discussion here does not have to do with the desirability 
of fidelity oriented implementation efforts. That is another topic. In this 
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paper the focus 1s on how to assess and judge Implementation success when It 
1s deemed Important to do so. 

A Framework for Defining Implementation Success 

Traditionally, at some point after an Innovation is introduced to 
potential users, an evaluation study 1s done to document that desired outcomes 
(e.g. Increased student achievement) have resulted. Other outcomes could be 
related to teacher morale, school climate, cost benefits and resource 
consumption. All of these either Implicitly or explicitly assume that use of 
the Innovation 1n appropriate ways has been accomplished by the prospective 
users. Diagrammatical ly this view of change would be: 

© 

Introduce j| » Assess 
Innovation ^ > Outcomes 

A more 1n-depth view of change would have several additional pieces: 

Introduce W > Monitor M > Certify Use M P Assess 
Innovation Implementation of Appropriate M P Outcomes 

Practices 

In this second approach assessing outcomes 1n order to make summative 
judgements about use of the Innovation would not be done until there was (1) 
documentation that Individuals (e.g. teachers) are users and (2) that what 
they are doing represents appropriate Innovation practices. When both have 
been documented then Implementation can be verified and summative questions 
about effects of use can be addressed. 

At this point 1t 1s Important to Identify the factors Involved 1n 
successful Implementation. Three diagnostic dimensions of the CBAM model 
provide the concepts and measures for defining Implementation success. In the 
CBAM approach Implementation Success (IS) is related to use/nonuse, 



appropriate/Inappropriate practice and user concerns about the innovation. 
The basic model Is: 

IS * f (LoU, IC, SoC) 

LoU 1s Levels of Use of the Innovation, IC 1s Innovation Configurations 
and SoC 1s Stages of Concern about the Innovation. Each of these vectors 
represent established concepts and measures for considering basic dimensions 
of Implementation as 1t occurs at the user level. These dimensions are 
briefly described in the appendix at the end of this paper. 

With these dimensions 1t 1s possible to obtain systematic data about each 
person's use of and concerns about an innovation. Progress can be measured in 
terms of" the Implementation of an Innovation for each of the three dimensions. 
When da ta from each d 1 men s Ion 1 s compa red for individuals or groups of staff 
1t 1s then possible to rank order subjects on a scale ranging from those who 
have moved the least or regressed to those who have advanced the mostln terms 
of mastering use of an Innovation. The rankings for each of the three 
dimensions can be combined to give a composite ranking of implementation 
success. These rankings can then be used as criteria for exploring the 
relationship between Implementation success and various change process 
variables. 

Following a brief description of the Principal -Teacher Intervention study 
will be a description of the procedure which was developed to assess 
Implementation success at a given point in time. In addition, the paper 
Illustrates how the procedure was applied using the PTI study data base and 
reveals some Illustrative findings of how Implementation success correlated 
with various change process variables. 



The Psr1nc1 pal -Teacher Interaction Study 

The Principal -Teacher Interaction Study focused specifically upon school 
principals to determine what they do to aid the Implementation of new programs 

i 

1n their schools. Three major questions guided this study: (1) What do. 
principals do as change facilitators? (2) How do the concerns of principals 
affect their functioning as change facilitators? and (3) What is the 
relationship between administrator concerns, the interventions they make and 
their effects on teachers? 

Nine elementary school principals served as the prima. y subjects and 
Informants on Interventions 1n this study 1n which data were collected from 
May 1980 to May 1981. Three principals were selected from school districts in 

Colorado, Florida and California. Based on composite descriptions by the 

researchers, the principals were chosen by district administrators to 
represent three approaches *o facilitating Implementation. Each of the three 
school districts were at different points 1n the process of implementing a 
district-wide Innovation. The California school district was beginning its 
first year of Implementation of a new writing composition program; the Florida 
district was beginning Its second year of implementation .. i a unified math 
curriculum; and the Colorado district was Into Its third year of 
Implementation of a revised science curriculum. So, the study sites varied by 
principal's approach to facilitation, year of Implementation, innovation being 
implemented, as well as by school district. 

Study Procedures 

Principals were trained on an Individual basis to identify and describe 
Interventions made by themselves and others. In these sessions, principals 
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were taught how to provide sufficient Information in reporting Interventions 
to allow for coding of the Interventions on various dimensions. 

Throughout the 1980-31 school year, researchers contacted study 
principals (and assistant principals) by telephone on a bi-weekly basis during 
the 1980-81 school year to collect data on interventions related to the 
innovations being studied. Intervention data were then coded by researchers 
on various dimensions Including sublevel, source, target, function, medium, 
flow and location. 

Three to four on-site visits were made to each study site. Intervention 
data were collected from administrators and teachers during these on-site 
visits. In addition data were collected from teach . a using the diagnostic 
dimensions of the CBAM. A summary and time line of data-collection procedures 
1s provided 1n Figure 1. Teachers completed the Stages of Concern 
Questionnaire four times during the study and were interviewed by a certified 
Levels of Use Interviewer three times during the year. Innovation 
Configuration information was collected through a "configuration hunt" at the 
beginning of each of the three Levels of Use Interviews. This information was 
then used to complete the IC checklist that had been developed for each site 
using established procedures (Heck, Stlegelbauer, Hall & Loucks, 1981). 

Since 1n all three districts the Innovations were locally developed or 
were local modifications of materials developed elsewhere, 1t was possible to 
go to the primary developers as key sources 1n the identification of 
configuration components and for setting criteria 1n terms of acceptable and 
unacceptable variations of components. 
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Figure 1 

PTI Data Collection Procedures and Timeline 



Principals May 1980 Oct. 1980 Jan. 1981 

(and Assistant Principals) 



May 1981 



Intervention phone 
Interviews 

Intervention face-to-face 
Interviews 

Field note-taking 

Context survey 

Change facilitator 
Stages of Concern 
Questionnaire 



bi-weekly 



X 
X 



X 
X 



X 
X 
X 



X 
X 



Teachers 

Intervention face-to- 
face interviews 

Stages of Concern 
questionnaire 

Levels of Use interview 

Innovation Configuration 
interview 

School .Climate Survey 



X 
X 



X 
X 



X 
X 



/ 



X 
X 



District Personnel 

Intervention phone 
interviews 

Intervention face-to-face 
Interviews 

Context Survey 



intermittent 
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Assessing Innovation Implementation Success: A New Procedure 

CBAM data that were gathered from teachers during the PTI study were the 
focus of project staff and representatives from each of the three research 
sites 1n a series of activities to analyze innovation implementation at the 
classroom and school 1 i. The task was to use each CBAM diagnostic 
dimension to rank order the nine study schools and then to combine the 
rankings to establish a measure of overall Implementation success. It would 
be possible then to correlate the degree of Implementation success with a 
number of variables Including change facilitator style of the principal. 

The ranking task was done during three three-day meetings by research 
staff and the central office representative from each school district who had 
been the liaison throughout the study. The task was done in three segments: 
the Interpretation and rankings of Stages of Concern and Levels of Use data 
were done in February 1982, the rankings of Innovation Configuration was 
accomplished in September 1982, and 1n November 1982 site representatives met 
again with project staff to examine the rank orderings and their relationship 
to change fad' 'tator^tyles and to the Initial cross tabulations of the 
Intervention data. 

Levels of Use Data 

Levels of Use was the first d1<nensicn used to rank order the schools. 

< 

The ten participants 1n the activity, each of whom had past experience 1n 
working with CBAM data, were given the LoU distributions for all data 
collection periods for each of the nine schools, but were not given the 
Identity of the schools. The task was twofold: 1) to rank the schools 1n 
terms of implementation success using, LoU as the criterion; and 2) to make 



notes on the Indicators being used as the basis for making the ranking. Each 
person did the task. Then the group met to summarize the Individual rankings 
i and derive a consensus ranking, 

There was strong agreement across the raters on the rankings and there 
was minimal expression,. 6f need to have any additional information with which 
to perform the ta^sx. The group did point out that the task would have been 
easier if information about the percentage of teachers at each LoU had been 
provided instead of the raw counts. Some of the factors mentioned by 
participants that Influenced their ranking were: amount of movement toward 
higher levels, proportion of -movement toward higher levels, size of school, 

/ 

/the final distribution, few persons at LoU Ill's (Mechanical Use) at the end, 
backward movement was a minus, and some favored having more IV B's. Nonuse to 
/ use was seen by some as more difficult to achieve than was movement from LoU 

III to higher LoU's and therefore was more heavily weighted. 

Each rater rank ordered the nine schools that were assigned the letters 
of A-I. An example of two raters' rankings 1s shown below. 



Rank Order Rater 1 Rater 2 Etc. 

1 D D 

2 A A 

3 G G 

4 B F 

5 F B 

6 H . C 

7 C H 
.8 I I 

9 E E 



The LoU data were summarized school by school by adding the rank order number 
assigned by each of the ten participants and deriving a total number. An 
example of how this procedure was conducted is shown below. 
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Rater 



School A 



School B 



Etc. 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 



2 
2 
1 
2 
3 
2 
2 
1 

4' 
2 



4 

5 
4 
3 
4 
4 
3 
4 
5 
3 



As can be seen 1n the examples above, when rater #1 rank ordered the nine 
schools, he ranked School A as second and School 6 as fourth. Rater #2 ranked 



School A as second and School B as fifth. By totaling the rank order numbers 
asslgne'd by all 10 raters, a total score was derived. Using this procedure, 
the lower the total score, the more successful the school's Implementation was 
judged 1n terms of the criterion being rated. The following total scores were 
derived for the nine study schools using the criterion of Levels of Use and 
are displayed by school ID number. 



Stages of Concern Data 

The task was repeated using the SoC profiles for each school. This task 
was slightly more difficult for mo. people but still quite possible. 



Implementation Success 
As Determined by Level of Use Data 



School 



Score 



11 
12 
13 
24 
25 
26 
37 
38 
39 



48 
58 
37 
26 
12 
82 
63 
51 
73 
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Instead of having a single score for each teacher like Lot), SoC data consists 
of several different points which make up a profile making comparison more 
difficult. In addition^ all of the intricacies of profile interpretations had 
to be considered. 

Again, reasonably strong agreement among raters was observed. Some of 
the Indicators used by participants 1n ranking the SoC data were: valuing 
resolution of early concerns, especially Stage 2 Personal; arousal of impact 
(Stages 4, 5 and 6) concerns. Tailing up on Stage 6 Refocusing was viewed 
negatively, as were unresolved Stage 3 Management concerns and little or no 
movement. The SoC profile at the final data collection time was also weighted 
heavily. The, SoC rankings were summarized using the same procedure used with 
the Loll data. The following total scores were derived using the criterion of 
Stages of Concern and are displayed by school ID number. 

Implementation Success 
As Determined by Stages of Concern Data 



School 


Score 


11 


68 


12 


29 


13 


61 


24 


34 


25 


22 


26 


60 


37 


75 


38 


25 


39 


73 



Innovation Configuration Data 

The ranking task continued at the September 1982 meeting. This time six 
staff members and the three site representatives analyzed the Innovation 
Configuration data. The task was first to rank the three schools within each 
district and then to rank all nine schools. The IC Component Checklist data 
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were displayed by school showing the percentage of teachers using each 

E. 

variation of each component. 

The procedure began w^ith the site person first reviewing the Innovation 
and the IC Component Checklist in terms of "critical components" and "ideal," 
"acceptable," and "unacceptable" variations. Each member of the group then 
rank ordered the three schools within the district. The procedure was then 
repeated for the other two sites. Following each ranking task, the 
participants discussed the activity and the indicators that they had used to 
do the task. On the second day of the September session, all nine sites were 
ranked 1n terms of Implementation success based on IC data. One result of 
extensive discussions was the Identification of a basic set of guiding 
principles for rank ordering schools using IC data. Basically, Implementation 
success was viewed as being comprised of two vectors, 1) the proportion of 
teachers whose use moved them across the minimally acceptable line 1n the 
shortest period of time, and 2) growth toward and above the minimum acceptable 
standard. Participants reported that rank ordering schools within the 
district as difficult, yet still possible. In terms of the number of 
variables Involved 1n ranking, Innovation Configurations proved to be the most 
complex given three different Innovations and IC Component Checklists that 
were not standardized. The varying degrees of clarity provided by the 
district 1n terms of what comprised successful implementation also contributed 
to the difficulty of the task. 

The IC rank order data generated by the nine participants in the task 
were summarized 1n the same fashion as the LoU and SoC data. The following 
total scores were derived for the nine study schools using the criteria of 
Innovation Configuration. 




Implementation Success 
As Determined by Innovation Configuration Data 



School 



Score 



11 
12 
13 
24 
25 
26 
37 
38 
39 



49 
47 
53 
47 
41 
53 
14 
23 
33 



Configuration Data Problematic . Compared to the LoU and SoC tasks, the 
overall IC rank ordering was more difficult and problematic. The concepts of 
SoC and LoU allow each respondent some latitude 1n defining the Innovation 
whereas, 1n the Innovation Configuration work, this Individual interpretation 
1s restricted. SoC and LoU are generic concepts describing the Individual 
whereas IC focuses on the specific Innovation and the ways each component of 
1t 1s made operational. It was difficult . to compare the different Innovations 
which had different numbers of components and variations, plus 1t was not 
possible to conduct a 'blind' activity as the checklist Itself 'gave away' 
district identity, and thus the year of Implementation it represented. 

Other complications developed from the fact that each district valued 
certain components of their Innovation more than others. Also, not all of the 
variations 1n each component of the checklist were mutually exclusive and 



therefore 1n some components one or more variations would be marked, further 
complicating comparison across sites. In one district especially, 
expectations were not made clear and each school within that district had to 



district focused on components of the Innovation that were later decided by 
central office personnel not to be district priorities. 



determine what the priorities would be. Some of the study schools within that 
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From all of these complications, a great deal was learned about 
Innovation Configuration checklist construction. At the time of designing an 
IC Component Checklist there 1s a clear need to know how the resultant data 
will be used. For example, 1f the checklist 1s to be used for cross-site or 

cross-innovation comparisons, 1t needs to be designed specifically to 

* 

accommodate these types. *)f analyses whereas 1f 1t 1s only to be used to help 
facilitators 1n a single site, it can be more descriptive and less 
standardized. For component checklists that are to be used for cross-site 
comparisons, It 1s essential that each component variation be mutually 
exclusive 1n order to prevent multiple ratings within a single component. 

In future studies, 1f sites are to be ranked, there is a strong need to 
determine and express the criteria for success at the beginning of a change 
effort. It 1s Important that critical components be Identified at the 
beginning and that clear criteria for success for each component be 
articulated. Though the IC data analyses task was problematic, the overall 
conclusion of the group was that the Innovation Configuration concept +s~eveTT 
more critical to understanding and facilitating successful Implementation than 
they had Initially realized. The Innovation specific focus 1n combination 
with the articulation of criteria for each component make the IC data very 
sensitive 1n reflecting effects of Interventions and local conditions. 

Overall Implementation Success 

A score for overall Implementation success was derived for each of the 
nine study schools by summarizing the scores of each of the three criteria 
used to rank order the schools. The following overall scores were derived for 
the nine study schools using the criteria of Levels of Use, Stages of Concern 
and Innovation Configuration and are displayed by school ID number. 
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School 


LoU Score 


SoC Score 


IC Score 


Overall Score 


11 


48 


68 


49 


165 


12 


58 


29 


47 


134 


13 


37 


61 


53 


151 


24 


26 


34 


47 


'107 


25 


12 


22 


41 


75 


26 


1 82 


60 


53 


195 


37 


63 


75 


14 


152 


38 


51 


25 


23 


99 


39 


73 


73 


33 


179 



Again, 1t 1s Important to remember that the lower the score, the greater 
the Implementation success as determined by criteria used 1n each rank 
ordering task. Therefore, school 25 was judged to have the greatest degree of 
overall Implementation success and school 26 was judged to have the lowest 
degree of Implementation success. 

Several trends can be noted 1n this set of data. An obvious one 1s that 

j 
j 

year of Implementation was set the determining predictor of Implementation 
success. The top ranked school (25) and the bottom ranked school (26) were 
from the same district and 1n the first year of Implementation. Al s^jiejther 



the 'district nor the Innovation 1s a controlling factor 1n determining 
Implementation success, as the scores do not "clump" according to district. 
An additional point demonstrated by these analyses 1s that Implementation 
success can be defined 1n terms of 1 teacher and Innovation use variables and 
that Implementation success can be compared across sites Involved with 
different Innovations, 



Correlating Implementation Success 
With Other Variables 



Since this procedure places the various dimensions of Implementation 
success on an Interval scale, 1t becomes possible to easily correlate these 
rankings with other variables. 
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A Pearson Product Moment Correlation was used to determine wi.c 
relationship between Implementation success and numerous variables in the PTI 
(Stud^ data set (see Hord, Hull rig & Stlegelbauer, 1983, for a description of 
the PTI data base). One Important analysis explored the relationship of 
Implementation success to principal Change Facilitator Style. Using other 
procedures the research staff assigned a score ranging from 0-100 to each 
principal based upon their Change Facilitator Style (see Hall and Rutherford, 
1983, for a complete description of the procedure). A number of significant 
correlations were discovered. Variables which correlated most highly with 
each \>f the dimensions of Implementation success and overall Implementation 

t" 

success are shown in the tables that follow. 



Table 1: Variables Correlating Most Highly 
With, Levels of Use Implementation Success 



Principal Style 

N of Incidents Targeted At Teachers 
1n Groups (Targets 3B and 4B) 



N of Incidents Occurring 1n Location 1C 
N of Incidents Occurring in Function 3D 



Correlation 
.91 



.76 
.58 
.55 



Probability 
.001* 



.009 

.051* 

.062 



Indicates statistical significance 



Table 2: Variables Correlating Most Highly 
W/ith Stages of Concern Implementation Success 

/ / 



- f 

N of Incidents Occurring In Location 1C 
N of Incidents Occurring In Function 3D 
Principal Style 



Correlation 
.69 
.52 
.47 



indicates Statistical significance 
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Probability 

»• * 
.019 

.074 
.103 



Table 3: Variables Correlating Most Highly 
With Innovation Configuration Implementation Success 



Correlation Probability 



Principal Iiftl dents Targeted At Teachers 

As Individuals .79 

PrlndpaMncidents Occurring 1n Function 4A .77 

Incidents Occurring 1n Function 4C .73 

Incidents Occurring 1n Function 4A .66 

N of Principal + Second CF Incidents .62 

N of Principal Inp1 dents .61 

Principal Incidents Occurring 1n Function 8 .60 



.006 

.007* 

.013' 

.028* 

.037* 

.042 1 * 

.043* 



Indicates statistical significance 



Table 4: Variables Correlating Most Highly 
With Overall Implementation Success 



Principal Style 

N of Incidents Occurring in Locution 1C 
N of Incidents Occurring 1n Function 4C 
Principal Incidents Occurring 1n Location 1C 
N of Incidents Occurrlrig in Function 3D 



Correlation 
.74 
.69 
.60 
.57 
.55 



Probability 
.011* 
.019 
.043* 
.055* 
.061 



indicates statistical significance 



Discussion of Findings From First Use of the Procedure 
for Assessing Implementation Success 



The preceding tables Indicate those variables that were found to 
correlate most highly with the dimensions of Implementation success. As was 
explained earlier, the lower the Implementation success score, the greater the 
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success. 4 Therefore, when the Implementation success score 1s correlated with 
various change process variables, most of the significant correlations are 
negative (I.e. the more that happened, the greater the Implementation success 
the lower the score for Implementation success). In order to not confuse the 
reader unnecessarily, the negative sign has been removed from the correlations 
1n the preceding tables. It 1s also Important to note that correlations do 
not Indicate cause and effect, only that there 1s a relationship between two 
variables. However 1n the data used here, there 1s a lot of detail known 
about the Implementation process that unfolded 1n each school. A discussion 
and Interpretation of the variables that correlated highly with each dimension 

of Implementation success and overall Implementation success w1U be made 1n 

i 

the following subsections. 
Levels of Use 

The most significant correlation with Levels of Use as a criterion for 
"implementation success (see Table 1) was principal Change Facilitator style. 
The principal CF style score was assigned by the research staff using a scale 
of 0-100 which represented the RespondeV-Manager-Initlator continuum (see Hall 
& Rutherford, 1983). This correlation Indicates that the more that the study 
principals functioned as an Initiator, the higher the Levels of Use. 

i 

The second most significant correlation dealt with. Interventions, being 
targeted at teachers 1n subgroups as opposed to teachers as Individuals or as 
a whole staff. This correlation suggests that working with teachers 1n 
specialized subgroups was associated wl^h-hlgheTTeveis of Use.. A significant, 
correlation was also found with the number of Incident Interventions that 
occur 1n location 1C, Implementation site— other. Many Interventions occurred 
In, locations such as the teacher's lounge, the resource center or the school 
library and 1t 1s hypothesized that these Interventions account for the 
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correlation between Levels of Use Implementation success and Incidents 
occurring in location 1C. A high, but not statistically significant, 

correlation was found between Lou implementation success and the number of 

II 

incidents occurring in function 3D, Consulting and Reinforcing — information 
sharing. Theory and practice would agree that providing teachers with 
information is helpful ;» this analysis tends to support this position. 

Stages of Concern 

The only variable to correlate significantly with stages of Concern as a 
dimension of implementation success (see Table 2) was the number of incidents 
occurring in location 1C, Implementation Site— other. Again, Interventions 
that were coded as 1C for the location are those that occurred at the school 
In some place other than the office or in classrooms. These include 
Interventions that occurred in the teacher's lounge, the resource center or 
the school library. A^rently 1n some more successful schools there were 
mo re interventions in some of these locations. High, but not significant 
correlations, were found between Stages of Concern implementation success and 
the number of inc. dents occurring in function 3D, Consulting and Reinforcing 
— information sharing, and the principal's change facilitating style. 
Providing information seems to be important 1n theory and in practice for the 
arousal and resolution of Stages of Concern. 

Innovation Configuration 

The variable correlating most significantly with Innovation Configura- 
tion implementation success (see Table 3) was the number of principal 
Incidents targeted at teachers as Individuals, as opposed to teachers as 
groups or as a whole. Significant correlations were also found with the 
number of principal Incidents occurring in function 4A, Monitoring and 



Evaluating — Information gathering, and 1n function" 8, Expressing and 
Responding to Concerns. Significant correlations were also found 1n the 
number of Incidents (all sources) occurring 1n functions 4A, Monitoring and 
Evaluating -- information gathering, 4C, Monitoring and Evaluating — 
reporting, the number of principal Incident Interventions and the number of 
principal plus second change facilitator incident Interventions. Most of the 
correlations suggest relationships that make sense 1n theory and practice. In 
combination they suggest that close monitoring feedback and doing more 
Interventions will help Implementation of more acceptable configurations of >n 
Innovation. 

Overall Implementation Success 

Principal change facilitating style was the variable that correlated most 

-significantly with-overall -Implementation success (see Table_ 4). This was a. 
key finding from the PTI study and further supports the hypothesis that 

_pjHnclpal .change facilitator style is a viable concept and that the 
principal's change facilitator style does significantly Influence the 
classroom practice of teachers (Hall & Rutherford, 1983). Other variables 
that correlated significantly were the number of Incidents (all sources) and 
the number of principal Incidents occurring in location 1C, Implementation 
Site— other. 

The frequency with which Location 1C appears as a significant variable 
suggest that 1n future studies additional locations at the school site should 
be assigned specific location codes. Some of those sites in need of a 
specific code Include the Teacher's Lounge, the Resource Center and the school 
library. 

Another variable correlating significantly with overall Implementation 
success was the number of Incidents occurring 1n function 4C, Monitoring and 
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Evaluating -- reporting. A high, but not significant, correlation was found 
between overall Implementation success and the number of Incidents occurring 

1n function 3D, Consulting and Reinforcing — Information sharing. Again, 

I i 

doing more interventions and monitoring with feedback seem to be important. 
However, the major finding continues to be the difference that the change 
facilitator style of the principal appears to make 1n influencing 
Implementation success. 

Summary 

Viewing Implementation success as a function of Levels of Use, Stages of 
Concern, and Innovation Configuration makes good conceptual sense 1n light of 
all that has been learned about the Implementation process. In the past, 1t 
has been problematic to 1) agree on what 1s meant by the term "Implementation 
success" and to find tools to assess Implementation success, 2) compare 
Implementation success across sites or across innovations, and 3) correlate 
Implementation success with other change process variables. 

The procedures described-; in this paper accomplish all three tasks. 
Further, 1t has been tested using actual data from the Principal -Teacher 
Interaction Study. The various dimensions of implementation success and 
overall Implementation success can be rated on an Interval- scale and thus can 
be correlated with other change process variables. When applied to actual 
data, the correlations reveal meaningful relationships. 

The procedure 1s practical, manageable and cost-efficient. For these 
reasons, the procedure promises to be a useful technique for the future study 
of school Improvement. 
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Appendix A 

The Three Diagnostic Dimensions of the Concerns Based Adoption Model 

The CBAM 1s based on several assumptions about change. First, change 1s 
not an event, but a complex process requiring a considerable Investment of 
time and energy. Second, each Individual experiences change 1n a different 
way, at a different rate, and with a different Intensity. Third, there are 
oas1c characteristics of the change process that apply to both process and 
product Innovations. These Involve Individual growth 1n two dimensions:, the 
concerns the Individual has about the change and how he or she uses the 

program, product, or 1deG/(1,e. Innovation) being Implemented. An additional 

ii / 

assumption 1s that when/those responsible for facilitating change (principals, 
deans, Staff developers) have Information about where Individuals are 1n th» 
process, they are better able to provide appropriate aid and support to each 
Individual as he or she needs It. The model Includes three diagnostic 
dimensions, Stages of Concern, Levels of Use and Innovation Configuration, 
each of which addresses a key vector of how Individuals are progressing 

) 

relative to/Implementation of a particular Innovation. 

/ 

Stages of Concern 

One^ major dimension of the CBAM focuses on the "concerns" of individuals 

/ 

as they are Involved 1n change. This dimension, Stages of Concern About the 
Innovation (SoC) (See Figure 2), describes seven kinds of concerns that 
Individual s experience with varying Intensities as they experience the change 
process. These range from early concerns about "self," to concerns about 
"task," and finally to concerns about "Impact." A reliable and valid 
Instrument for measuring Stages of Concern, the SoC Questionnaire, as well as 
methods for Interpreting the measures, (Hall, George & Rutherford, 1977), have 
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Figure 2 

STAGES OF CONCERN ABOUT THE INNOVATION* 



STAGES OF CONCERN 


DEFINITIONS 


r 
i 

M 
P 


6 


REFOCUSING 


The focus 1$ on exploration of more universal benefits from 

thi* innovation irieludinct the nossiSilitv of maior chanae? 

or replacement with a more powerful alternative. Indi- 
vidual has definite Ideas about alternatives to the proposed 
or existing form of the Innovation. 


A 
C 


5 


COLLABORATION 


The focus 1s on coordination and cooperation with others 

rPnarH inn uc* nf th* innovation 

1 CUOl U Illy U)C wl bllC 1 IIIIUVQV 1 WJI . 


L • 

T 


CONSEQUENCE 


Attention focuses on Impact of the Innovation on student 1n 
his/her Immediate sphere of Influence. The focus Is on 
relevance of the Innovation for students, evaluation of 
student outcomes ineludina oerformanee and enmneftencies 

and changes needed" to .Increase student outcomes'. - / 


K 


MANAGEMENT 


Attention Is focused on the processes and tastes of using 
the Innovation and the best use of Information and resources. 
Issues related to efficiency, organizing, managing, sched- 
uling, and time demands are utmost. 


< 
I 


~ 2 

• 

m • 


PERSONAL 


Individual 1s uncertain about the demands of the Innovation, 
his/her inadeauacv to meet those demands, and his/her role 
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with the Innovation. This Includes analysis of his/her 
role 1n relation to the reward structure of the organiza- 
tion, decision making, and consideration of potential con- 
flicts with existing structures or personal commitment. 
Financial or status Implications of the program for self and 
colleagues may also be reflected. 


I 
f 


. 1 

1 


INFORMATIONAL 


A general awareness of the Innovation and interest in learn- 
ing more detail about it is Indicated. The person seems to 
be unworrled about himself/herself 1n relation to the inno- 
vation. She/he 1s Interested In substantive aspects of the 
Innovation In a selfless manner such as general character- 
istics, effects, and requirements for use. 




0 


AWARENESS 


Little concern about or Involvement with the Innovation 1s 
Indicated. 



Hall, G, E., George, A. A.< * Rutherford, W. L. Measuring stages of concern 
about the Innovation: A manual for use of the SoC questionnaire. Austin 
Research and Development Center for Teacher Education, The University of 
Texas, 1977. 
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been developed and applied 1n a number of cross-sectional and longitudinal 
studies (Hall, 1976; Hall & Rutherford, 1976; Loucks, 1977; Huling, 1981). 

The SoCQ consists of 35-1tems which the respondents rate by using an 
eight point Ukert scale. There are five items representing each of the seven 
Stages of Concern. Estimates . of Internal reliability (alpha coefficients) 
range from .64 to .83, Test-retest correlations range from .65 to .86. 
Percentile tables have been established for converting the raw scale scores 
and Interpretation procedures have been developed for both quantitative and 
qualitative analyses. The most useful Interpretations are derived from 
analysis of the profiles that are made from displaying the percentile values 
for each scale on a grid. The various analysis and interpretation procedures 
can be applied to Individual and group data. 

A further strength of the SoCQ is that the questionnaire Items are 
written so that they can be applied to any educational innovation. The Items 
remain t'ne same, thereby preserving the factor structure. The only change 
that 1s required for different applications 1s to change the name of the 
innovation on the cover page. 

Levels of Use 

The second diagnostic dimension of the CBAM, Levels of Use of the 
Innovation (LoU) (See Figure 3), describes how performance changes as the 
Individual becomes more familiar with an Innovation and more skillful at using 
1t. The Stages of Concern dimension focuses on perceptions of feelings about 
the Innovation. Levels of Use focuses on whether or not the teacher 1s using 
an Innovation. Eight distinct Levels of Use have been Identified (Hall, 
Loucks, Rutherford & Newlove, 1975). Typically an Individual begins with LoU 
0 "nonuse" of the Innovation, then moves to LoU I "orientation" about the 
Innovation and LoU II "preparation" for use. Initial use 1s usually at LoU 
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Figure 3 



LEVELS OF USE OF THE INNOVATION 



LEVELS OF USE : " DEFINITION OF USE 



0 NONUSE STATE IN WHICH THE USER HAS LITTLE OR NO KNOWLEDGE OF 

THE INNOVATION, NO INVOLVEMENT WITH THE INNOVATION, 
AtlO IS DOING NOTHING TOWARD BECOMING INVOLVED. 

Decision Point A .exit action to lum mm detailed information about tht 
Innovation. 



I ORIENTATION 

•! • 



STATE IN WHICH THE USER HAS RECENTLY ACQUIRED OR IS 
ACQUIRING INFORMATION ABOUT THE INNOVATION AND/OR 
HAS RECENTLY EXPLORED OR IS EXPLORING ITS VALUE 
ORIENTATION AND ITS DEMANDS UPON USER AND USER 
SYSTEM. 



Decision Point I Makes a decision to use the Innovation by establishing a 
t1»e to begin. 

II PREPARATION state in which the user is preparing for first use 

OF THE INNOVATION. 

Decision Point C Changes, If any, and use are dominated by user needs. 

Ill MECHANICAL USE state in which the user focuses most effort on the 

SHORT-TERM, DAY-TO-DAY USE OF THE INNOVATION WITH 
LITTLE TIME FOR REFLECTION. CHANGES IN USE ARE 
MAOE MORE TO MEET USER NEEDS THAN CLIENT NEEDS. 
THE USER IS PRIMARILY ENGAGED IN A STEPWISE ATTEMPT 
TO MASTER THE TASKS REOUIRED TO USE THE INNOVATION, 
OFTEN RESULTING IN DISJOINTED AND SUPERFICIAL USE. 

Decision Point D-I A routine pattern of use Is established. 

IVA ROUTINE use of the innovation is stabilized, few, if any, 

CHANGES ARE BEING MADE IN ONGOING USE. LITTLE PREP- 
ARATION OR THOUGHT IS BEING GIVEN TO IMPROVING INNO- 
VATION USE OR ITS CONSEQUENCES. 

Decision Point D-2 Changes use of the Innovation based on formal or Informal 
evaluation 1n order to Increase client outcomes. 

IVB REFINEMENT state in which the user varies the use of the inno- 
vation TO INCREASE THE IMPACT ON CLIENTS WITHIN THE 
IMMEDIATE SPHERE OF INFLUENCE. VARIATIONS ARE BASED 
ON KNOWLEDGE OF BOTH SHORT- AND LONG-TERM CONSEQUENCES 
FOR CLIENTS. 

Decision Point E Initiates changes In use of Innovation based on Input of and 
1n coordination with what colleagues are doing. 

V INTEGRATION state in which the user is combining own efforts to 

USE THE INNOVATION WITH RELATED ACTIVITIES OF COL- 
LEAGUES TO ACHIEVE A COLLECTIVE IMPACT ON CLIENTS 
WITHIN THEIR COMMON SPHERE OF INFLUENCE. 

Decision Point F Begins exploring alternatives to or major modifications of 
the innovation presently 1n use. 

VI RENEWAL state in which the user reevaluates the quality of 

USE OF THE INNOVATION, SEEKS MAJOR MODIFICATIONS OF 
OR ALTERNATIVES TO PRESENT INNOVATION TO ACHIEVE 
INCREASED IMPACT ON CLIENTS, EXAMINES NEW DEVELOP. 
ME NTS IN THE FIELD, AND EXPLORES NEW GOALS FOR SELF 
AND THE SYSTEM. 
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From: The LoU Chart. Austin: Research and Development Center for Teacher 
Education, The University of Texas, 1975. 

ttm Project 

Research and Development Center for Teacher Education 
The University of Tews 0 Q 



Ill "mechanical ," but as experience Increases, Innovation users move to a LoU 
IVA "routine" level of use and eventually may reach various "refinement" 
levels (LoU IVB - VI), where changes are made based on formal or Informal 
assessments of student needs. A focused Interview procedure has been 
developed to measure Levels of Use (Loucks, Newlove & Hall, 1976) and has been 
used 1n a number of research and evaluation studies (Hall & Loucks, 1977a; 
Hall & Loucks, 1977b; Loucks, 1976). 

'* The LoU Interview takes fifteen to thirty minutes and 1s generic 1n' 
design, as 1s the SoCQ, so that 1t can be applied to different Innovations 
with no change In the basic Interviewing process. When conducted by trained 
and certified LoU Interviewers the Interview process appears to the 
Interviewee as an interested and comfortable conversation about what they are 
doing or not doing with the Innovation. The Interviewer uses the, LoU concept 
and operational definitions along with a set of decision points to conducts 
focused Interview that branches 1n varying ways as more 1s learned about what 
the Interviewee 1s doing. These Interviews are tape recorded, so as to be 
available for further analyses. The Interviewee is rated as performing at one 
of the Levels of Use. The Individuals rated can be displayed 1n a table that 
reflects the number of persons that were Identified at each LoU. 

Training procedures and a series of steps that lead to certification as a 
valid and reliable LoU Interviewer, have been established by R&D Center staff. 
Inter-rater reliabilities typically range from .87 to .96 and 66 percent 
agreements between the first two raters is commonly observed. 



Innovation Configurations 




The third diagnostic dimension of the CBAM that 1s Important in 
understanding and describing the change process 1s Innovation Configurations 
(IC) (Hall & Loucks, 1978; 1981). This concept 1s used to describe the 
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- — — various operational forms of an Innovation that result as users adapt it for 

use 1n their particular situations. With this concept, the major operational 
components of an Innovation are Identified and the ways that each of the 
components can vary are described. These descriptions are summarized on an 
Innovation Configuration Components Checklist. The IC Component Checklist 1s 
Innovation specific and can be used to record 1n what ways each potential user 
1s using the various parts of the Innovation. 
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